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ABSTRACT 

We  give  a  combinatorial  definition  of  the  notion  of  a  simple 
rectilinear  region  being  it-concave  where  Jfc  is  a  non-negative 
integer.  The  usual  definition  of  rectilinear  convex  is  1-concave 
in  our  sense.  Then  we  consider  the  pjroblera  of  computing  the 
largest  rectilinear  convex  region  contained  in  a  general 
rectilinear  region.  An  0(n?)  algorithm  is  presented. 
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1.  Introduction 

The  general  problem  is  the  following:  given  a  simple  polygon,  find  the  largest  area 
convex  polygon  contained  in  it.  J.  Goodman  [Goo]  called  this  the  'potato  peeling  problem'; 
he  also  posed  the  question  of  finding  a  fmite  algorithm  for  the  problem.  Independently, 
[Woo]  considered  the  problem,  dubbing  it  the  'convex  skull  problem'.  Recoitly,  [ChY]  gave  a 
finiteness  criteria  for  the  largest  convex  included  polygon  and  solved  the  problem  in 
©(n'log  n)  time.  In  this  report,  we  investigate  the  rectilinear  versions  of  this  problem.  Note 
that  the  finiteness  of  the  convex  skull  problem  in  the  rectilinear  case  is  easy. 

2.  DeflnHkin  of  Jtr-concavtty  for  Rectilinear  Regtons 

By  a  (rectilinear)  region  we  mean  a  connected,  dosed,  bounded  subset  of  the  plane 
whose  boundaries  are  composed  of  line  segments  parallel  to  the  two  axes.  Note  that  the 
boundary  path  of  a  region  is  not  necessarily  self-avoiding  or  connected,  so  holes  are  allowed. 
To  avoid  degeneracies,  we  assume  that  a  region  is  equal  to  the  closure  of  its  interior.  If  the 
interior  of  a  region  has  no  holes  then  we  call  the  region  simple.  The  boundary  of  a  simple 
region  is  a  dosed  connected  path.  Note  that  the  path  may  not  be  self-avoiding:  there  may  be 
'kissing  comers'.  (Remark:  it  is  possible  to  define  'regions'  in  terms  of  'polygons'  except  that 
this  would  be  rather  more  complicated  since  the  boundary  of  our  regions  can  have  several 
components  that  are  not  necessarily  simple  polygons.) 

The  usual  definition  of  convexity  dearly  implies  that  rectangles  are  the  only  rectilinear 
regions  that  are  convex.  However,  less  restrictive  definitions  are  also  known:  for  instance, 
following  [OSW],  define  a  rectilinear  region  to  be  rectilinear  convex  if  its  intersection  with  any 


horizontal  or  vertical  line  is  a  (possibly  empty)  connected  line  segment.  We  provide  an 
alternative  approach  to  defining  convexity  by  considering  the  more  general  problem  of  trying 
to  qxiantify  the  'degree  of  concavity'.   Indeed,  we  will  define  the  notion  of  'it-concavity'  (for 

it  =  0,1,2 )    such   that   rectangles   correspond   precisely   to   0-concave   regions,    and 

rectilinear  convex  regions  to  1-concave  regions.  Oiir  definiton  of  Jt-concavity,  though  quite 
natiiral,  is  non- trivial  to  make  precise.  We  remark  thiat  a  completely  different  approach  to 
quantifying  concavity  is  used  in  [Skl72]  where  the  area  of  'concave  pockets'  of  a  region  is 
compared  to  the  total  area  of  the  region;  in  contrast,  our  approach  is  strictly  combinatorial. 

Let  /?  be  a  simple  region  represented  by  its  the  vertices  (vg,  vj v„_,,  v„)  on  its 

boundary  path,  listed  in  clockwise  order.  In  this  list,  vg  =  v„  and  also  v^  =  Vj  for  other  values 
of  i  ^  ;  whenever  the  path  kisses  itself  at  v,,  so  there  are  only  ^  n  distinct  vertices.  We 
assign  a  turning  number  Ty  to  each  v,  such  that  Tq  =  0  and 


T/+1   = 


T(  + 1  if  V;  is  a  right  turn 
T|  - 1  if  V(  is  a  left  turn. 


In  particular,  if  /?  is  a  rectangle  the  sequence  of  turning  numbers  is  (tq, 
3,  4).   As  another  example,  suppose  A  is  the  following  U-shape: 


tJ  =  (0,  1,  2, 
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Figure  1.  U-shape 

Then  the  turning  sequence  is  (tq,  .  ,  .  ,  Tg)  =  (0,1,2,1,0,1,2,3,4).  Qearly  the  turning 
numbers  as  defined  depend  on  the  particular  choice  of  initial  vertex  vg.  One  invariant 
property  of  these  turning  numbers  is: 


Lemma  1.  t„  =  4  . 

Proof.  This  is  equivalent  to  the  more  general  claim  that  t„  —  Tq  =  4  for  any  turning 
sequence  where  the  initial  value  Tq  can  be  any  integer.  We  prove  the  more  general 
formulation  by  induction  on  n.  Choose  the  coordinate  system  such  that  the  two  edges  incident 
at  Vq  are  on  the  positive  x-  and  y-eaes.  We  may  assume  n  >  4.  Suppose  that  R  does  not  lie 
entirely  in  the  first  quadrant.    Choose  «•  to  be  the  point  on  the  boundary  of  R  such  that  the 


horizontal  segment  [v„_j,  «♦]  or  the  vertical  segment  [vj,  «*]  separates  R  into  two  (non- 
degenerate)  regions  R^,  /?2-  (At  least  one  of  these  two  possibilities  must  hold.)  Wlog,  assume 
that  R  is  separated  by  [vj,  u*]  into  R^  to  its  left  and  /?2  to  its  right.  Write  the  turning  number 
of  a  vertex  v  in  *  (resp.  R^,  /?,)  as  t(v)  (resp.  (t(v),  s'(v)).  Here  we  assume  the  initial  vertex 
of  R  (resp.  R^,  R^  is  vq  (resp.  v^,  Vq)  with  initial  value  t(vq)  =  0  (resp.  ct(v])  =  -1, 
<y'(vQ)  =  0).  Note  that  this  is  unambiguous  provided  we  agree  to  distinguish  the  multiple 
appearance  of  a  point  on  the  boundary  path  of  a  region.  By  the  induction  hypothesis  for  R., 
if  V*  is  the  node  of  /?,  preceding  u*  then  cr(v*)  =  1.  But  note  that  t(v*)  =  ct(v*).  Also,  if 
w*  is  the  node  of  /?,  following  u*  then  (t'(w*)  =  cr'(w*)  ±  1  =  1±  1  (depending  on  the 
nature  of  the  turn  at  w*).  Therefore  cr'(>v*)  =  t(h'*).  Thus  t(v)  =  a'(v)  for  all  v  following 
w*  and  in  particular  t  and  ct'  agrees  on  their  last  vertex.  Hence  by  the  induction  hypothesis 
for  /?■),  we  conclude  that  t(v„)  =  4,  as  required.  The  other  cases  where  R  is  confined  to  the 
first  quadrant  can  be  similarly  proved  by  choosing  u*  such  that  the  vertical  segment  [v„_2,  u*] 
or  the  horizontal  segment  [v2,  «*]  separates  R  into  two  regions.   Q.E.D 

We  now  turn  to  another  invariant  of  these  numbers.    For  i  =  l n-1,  we  call  i  a 

bcal  maxima  (resp.  minima)  if  t,  >  T(_i  =  t,+i  (resp.  t,  <  t,_j  =  t^+j).  We  extend  this 
definition  to  i  =  0  by  defining  0  to  be  a  local  maxima  (resp.  minima)  if  Tq  >  Tj  =  t„_i  -  4 
(resp.  Tq  <  t„_j  -  4).  Note  that  this  definition  makes  sense  because  of  the  previous  lemma. 
Fmally,  let 

Tjjjjj,  =  raax{T,  :  I  =  0,  1 n-1   is  a  local  maxima  } 

T^n  =  min{T,  :  J  =  0,  1,  .  .  .  ,  n-1   is  a  local  minima  }. 
If  there  are  no  local  maxima  (resp.  minima)  then  define  r,^^  =  0  (resp.  t^„  =  0). 

Lemma  2. 

i)       /?  is  a  rectangle  iff  t,^  =  0  iff  t^„  =  0. 

ii)      If  /?  is  not  rectangular  then  r^^^  >  t^„  and  the  value  of  t„j^  -  t^„  is  a  property  of  R 
that  does  not  depend  on  the  choice  of  vq. 

Proof.  We  only  show  non-trivial  part  ii).  First  we  prove  that  t,^,^  >  t^,,.  Qearly  R 
has  some  local  maxima  and  local  minima.  Furthermore,  these  maximas  and  minimas  are 
alternating  in  the  sense  that  if  i\  <  {'2  are  two  local  maximas  (minimas)  then  there  exists  a 
local  minima  (maxima)  j  with  «\  <  j  <  »2-  One  can  further  show  that  the  number  of  maximas 
and  the  number  of  minimas  are  equal.  So  suppose  wlog 

0  s  i-j  <  y-j  <  »2  <  ;2  <  •  •  •  <  ik  <  A  <  « 

where  the  j/s  (resp.  y'/s)  are  local  maximas  (resp.minimas).    Let  t,  =  t,,,,,,  for  some 


h  =  1.  .  .  .  ,  k.  Then  we  have  the  desired  inequality 


'k 


max* 


Next   we   prove   that  t„^  -  t^^   is   a   property   of  R.     Let   (tq,  Ti,  .  .  .  ,  t„)    and 

(t'q,  t'i t'„)  be  two  sequences  of  turning  numbers  of  R,  with  respect  to  different 

choices  of  the  initial  vertex.   It  is  not  hard  to  see  that  for  some  h  ^  0, 


T  ;   = 


T,+h  -  T^  if  J  =  0,  1 n-h-l 

Ti^H-n  -  Ta  +  4    if  I  =  n-h n. 


Therefore,  the  local  extremes  of  the  two  sequences  (t,)  and  (t'J  are  in  one-one 
correspondence  and  it  can  be  verified  that  t',j^  -  t'jjj„=  t,,^  -  j^^.   Q.E.D 

In  view  of  this  lemma,  we  define  the  concavity  of  i?  to  be  t,,^  -  t^„.  A  region  is  called 
k-concave  if  its  concavity  is  at  most  k.  We  will  now  characterize  the  Jt-concave  regions  for 
it  s  2.   A  rectangular  region  is  naturally  represented  by  four  parameters: 

nan'     nax>  ^nari'  yirzx) 

We  call  the  rectangle  semi-infinite  if  exactly  one  of  these  four  parameters  is  infinite.  So  semi- 
infinite  rectangiilar  regions  are  essentiaUy  3-sided  figures.  The  relatively  easy  proof  of  the 
following  is  omitted: 

Theorem  3, 

a)  The  0-concave  regions  are  precisely  the  rectang\ilar  regions. 

b)  The  1 -concave  regions  are  precisely  the  rectilinear  convex  regions. 

c)  The  2-concave  regions  are  precisely  those  connected  regions  each  having  the  form 

k 

R  -  interior {\JS,)  ,      k^O 


where  /?  is  a  rectangular  region,  and  the  5/8  are  semi-infinite  rectangular  regions  whose 
interiors  are  pairwise  disjoint,  o 

Note  that  strictly  speaking,  the  rectangle  R  in  (c)  of  the  theorem  is  not  necessary:  2- 
concave  regions  can  also  be  characterized  as  the  bounded  connected  sets  that  are  complements 
of  a  finite  union  of  semi-infinite  rectangles.  The  next  figure  illustrates  a  2-concave  region. 


Figure  2.  A  2-ccmcave  region 


3.   Largest  Rectangular  Region  contained  In  a  Region 

It  should  be  dear  from  the  definition  that  the  concavity  of  R  can  be  determined  in  linear 
time.  Furthermore,  the  largest  rectangular  (i.e.  0-concave)  region  contained  in  R  can  be 
found  using  the  well-known  'plane-sweep'  technique  in  time  0{n  log  n).  The  more  interesting 
question  is  to  ask  for  the  largest  1 -concave  region  contained  in  R.  This  will  be  the  subject  in 
the  rest  of  this  paper. 


4.  The  Manhattan  Skyline  Problem 

To  begin,  we  consider  a  simple  case  that  will  give  some  insight  into  the  general  problem. 
Let  Rq  be  a  rectilinear  region  that  is  a  histogram  or  a  'Manhattan  skyline'.  It  is  not  hard  to 
verify  that  Rq  is  2-concave  (using  our  characterizaton  in  the  last  theorem). 


Figure  3.  Manhattan  Skyline 


Theorem  4.  A  largest  1 -concave  subregion  of  a  Manhattan  skyline  can  be  found  in 
0(n  log  n)  time. 

Proof.  Partition  the  skyline  Rq  into  rectangles  by  introducing  the  minimal  number  of 
horizontal  line-segments.  It  is  easy  to  characterize  these  segments  -  each  segment  may  be 
regarded  as  the  extension  of  some  horizontal  edge  of  the  skyline.  It  is  easy  to  see  that  there 
are  0(n)  horizontal  segments  and  hence  0(n)  rectangles  (called  slabs)  in  the  partition. 


Figure  4.  Segmenting  the  Skyline 


We  can  form  a  slab  tree  where  each  nrJe  corresponds  to  a  slab,  the  root  of  the  tree 
corresponding  to  the  (uniqxie)  bottori-uost  slab  and  the  children  of  a  node  v  consists  of  all 
those  nodes  whose  slab  sits  directly  on  the  slab  corresponding  to  v.  (The  above  illustration 
should  make  the  intent  clear.) 


We  associate  a  Veight'  with  each  node  of  the  tree:  the  weight  of  a  leaf  is  the  area  of  the 
slab  it  represents.  The  weight  of  each  internal  node  is  the  sum  of  the  area  of  its  own  slab  and 
the  maximum  of  the  weights  of  its  children.  The  weight  of  the  root  gives  the  maximum  area 
of  a  convex  subset  of  Rq.  The  computational  effort  can  be  seen  to  be  0(n  log  n).   Q.E.D 


5.   Maximal  1 -concave  subreglons 

For  brevity,  a  sub-region  of  a  region  always  mean  a  subregion  of  the  region  that  is  1- 
concave.  (For  emphasis  we  may  still  refer  to  a  '1-concave  subregion'.)  A  subregion  is  maximal 
if  it  is  not  properly  contained  in  another  subregion,  and  it  is  maximum  if  no  other  subregion 
has  a  strictly  larger  area.  (Thus  'maximal'  means  locally  maximum)  We  now  show  two  simple 
properties  of  maximal  subregions  that  can  be  exploited  algorithmically. 

Let  D  i  {N,  S,  E,  W}  he  any  of  the  four  compass  directions,  and  let  e  be  any  edge  of  a 
region  R.  If  the  vector  normal  to  e  and  outward  pointing  from  i?  is  in  the  direction  D ,  then 
we  say  tf  a  D-edge.  For  directions  C,D  ^  {N,  S,  E,  W),  the  common  endpoint  of  a  C-edge 
and  a  D-edge  is  caUed  a  CD-comer.  Two  directions  C  and  D  are  called  opposite  if  either 
{C,D)  =  {N,  S]  ox  {C,D]  =  {£,  W};  if  they  are  not  opposite  directions  then  they  are  adjacent 
directions.  Note  that  there  are  no  CD-comers  if  C  and  D  are  opposite;  conversely  if  C  and  D 
are  adjacent,  then  each  region  has  at  least  one  CD-comer.  (Note:  It  is  possible  for  edges  of 
all  four  directions  to  meet  at  a  point.  In  this  case  we  may  want  to  define  the  notion  of  CD- 
comers  more  carefully;  this  is  not  needed  here  and  we  leave  it  to  the  reader.)  A  comer  is 
convex  if  a  sufficiently  small  disc  about  it  intersects  the  region  in  a  convex  set. 

For  example,  the  Manhattan  skyline  in  figure  3  has  1  S-edge,  14  A^-edges,  8  W-edges, 
and  7  £-edges;  it  also  has  8  convex  ATV-comers  and  7  non-convex  AW-comers,  and  a  unique 
convex  5>V-comer.  < 

We  call  an  edge  e  extremal  if ,  in  a  clockwise  traversal  of  the  boundary  of  R,  the  turning 
number  increases  both  times  that  we  pass  the  endpoints  of  e.  Alternatively,  if  «  is  an 
extremal  £-edge,  we  call  it  the  'easternmost'  edge;  and  similarly  for  'northernmost',  etc.  For 
instance,  the  Manhattan  skyline  example  has  5  northernmost  edges  and  1  extremal  D-edge  for 
the  each  of  the  other  three  directions  D.  It  is  not  hard  to  verify  that  a  region  is  1-concave  if 
and  only  if  it  has  a  unique  extremal  D-edge  for  each  D  €  {N,  S,  E,  W}.  If  e^,  and  e^  are  the 
extremal  N-  and  £-edges  of  a  1-concavc  region  then  the  boundary  from  e-^  to  e^  is 
monotonically  nondecreasing  in  the  southern  and  eastern  directions.  We  call  this  portion  of 
the  boundary  (not  including  e^/  and  e^)  the  NE-staircase.  Note  that  the  ^£-staircase  is  empty 
if  and  only  if  the  northernmost  and  easternmost  edges  meet,  if  and  only  if  thf.  NE-oomex  is 
unique.  Similarly  we  can  define  the  5W-staircase,  etc.  Therefore,  the  boundary  of  a  1- 
concave  region  can  be  decomposed  into  the  foUowing  parts:  the  four  extremal  edges,  and 
between  any  two  adjacent  extreme  edges,  the  'staircase'  connecting  them.  The  next  figure 
illustrates  these  definitions. 
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NW 
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Figure  5.  A  1 -concave  region  with  :mpty  5£-staircase 

Two  line  segments  are  said  to  overlap  if  their  intersection  is  a  line  segment  of  positive 
length. 

Lemma  5.  Let  /?i  be  a  majdmal  (1-concave)  subregion  of  Rq  and  e  be  any  edge  of  R^. 

i)       The  edge  e  must  overlap  some  edge  of  Rq. 

ii)      For  an  edge  /  of  Rq  that  overlaps  e,  e  {&&  D-edge  of  R^  iff  /  is  a  D-edge  of  Rq. 

iii)     If  tf  is  extremal  then  e  is  contained  in  some  (not  necessarily  extremal)  D-edge  of  Rq. 

The  proof  of  this  lemma  is  easy:  (ii)  is  immediate.  If  either  (i)  or  (iii)  is  violated,  we 
can  extend  /?i  to  a  strictly  larger  subregion.  This  simple  lemma  can  be  the  basis  of  an 
0{tr'  log  n)  algorithm  for  finding  a  largest  1-concave  subregion  of  a  region  R^.  for  each 
quadruple  (^v,  's.  <?£.  «w/)  of  edges  of  Rq,  where  each  ^^  is  a  D-edge  of  Rq,  find  in  0{n\og  n) 
time  the  maximal  1-concave  subregion  /?j  of  ^q  such  that  the  extremal  D-edge  of  /?j  is 
contained  in  e^,  for  D  €  {N ,  S,  E,  W};  this  is  done  by  constructing  in  0(nlog  n)  time  the 
'best'  staircase  between  ^^  s^d  e^  for  each  pair  of  adjacent  directions  C,  D.  This  can  be  done 
using  the  plane-sweep  technique,  but  the  details  may  be  left  to  the  reader.  In  the  next  section 
we  improve  this  simple  solution. 


6.   Algorithm  for  largest  1 -concave  subregioa 

Let  /?o  be  an  arbitrary  but  fixed  region  in  this  discussion.  We  now  describe  an  algorithm 
to  compute  the  area  of  the  maximum  subregion  of  the  region  Rq.  It  should  be  relatively 
straightforward  to  convert  our  description  to  an  algorithm  that  actually  computes  a  maximum 
region. 

As  in  section  4,  we  introduce  a  minimal  number  of  horizontal  line  segments  sufficient  to 
partition  Rq  into  rectangles  that  we  call  slabs  (of  R().  A  slab  B  is  immediately  below  another 
B'  if  the  5-side  of  B'  overlaps  the  W-side  of  B.  The  reflexive  transitive  closure  of  this  relation 
is  the  below  relation.   The  inverse  of  the  '(immediately)  below*  relation  is  the  '(immediately) 


10 


above'  relation.  Again,  we  can  form  a  directed  graph  with  nodes  representing  the  slabs  and 
directed  edges  from  each  slab  to  those  slabs  immediately  below.  This  digraph  is  a 
generalization  of  the  slab  tree  previously  defined  for  the  Manhattan  skyline.  Our  algorithm 
will  process  one  slab  at  a  time,  starting  from  slabs  that  are  not  below  any  other  slabs,  in  the 
order  of  any  topological  sort  of  the  digraph. 

A  simple  but  useful  concept  is  that  of  'illumination'.  Let  X  be  any  set  of  points.  A 
point  p  is  said  to  be  illuminated  by  X  (from  below)  if  there  exists  a  point  q  i  X  vertically  below 
p  such  that  the  line-segment  [p,  <?]  lies  in  /Jq.  We  can  similarly  talk  of  p  being  illuminated 
from  the  west  if  ^  is  west  oi  p,  etc.  Typically,  X  =  5  is  a  slab  and  the  part  illuminated  by  X 
from  below  is  thus  a  Manhattan  skyline,  denoted  Mg,  with  B  QMg  a&  its  'base'.  Now  let  B 
be  fixed  and  consider  a  point  p  \b  m  the  boundary  of  A/3.  If  />  is  not  in  the  boundary  of  Rq 
then  p  must  be  in  some  eastern  or  western  edge  oi  Mg.  A  maximal  connected  set  of  such 
points  forms  a  line-segment  called  a  window  of  Mg.  The  window  is  either  eastern  or  western 
depending  on  the  edge  of  Mg  that  it  is  contained  in.  Let  w  be  any  window.  A  subregion  R  is 
said  to  abut  a  window  w  if  the  easternmost  or  westernmost  edge  of  R  overlaps  w.  The  non- 
illuminated  component  of  Rq  that  abuts  a  window  w  is  called  the  pocket  outside  the  window  w. 

Let  A/  C  J?o  be  a  Manhattan  skyline  obtained  by  illumination  from  its  base  B .  We  will 
associate  an  induced  slab  tree  Tg  with  M  where  the  induced  slab  tree  is  similar  but  not 
necessarily  identical  to  the  slab  tree  associated  with  M  in  section  4.  Pi'ecisely,  Tg  is  induced  by 
Rq  in  the  sense  that  each  slab  of  Tg  has  the  form  MHS  where  5  is  a  slab  of  Rq.  Therefore  Tg 
is  in  general  a  refmeraent  of  the  slab  tree  defmed  for  M  in  section  4.  It  is  convenient  to  speak 
of  a  node  of  Tg  interdiangeably  with  the  slab  it  represents.  With  each  node  v  in  Tg,  we  store 
the  following  values 

a(v),  3(v), 

and 

€,(v),  c.,(v),   (i  =  0,  1,  2,  3) 

defined  as  follows: 

(1)  An  eastern  flank  of  v  is  a  subregion  R  Q  M  such  that  R  has  unique  NW-,  SW-,  and  SE- 
comers,  with  the  MV-comer  lying  in  the  ray  projecting  vertically  downwards  from  the 
5£ -comer  of  v.  f^(v)  is  the  area  of  the  largest  eastern  flank  of  v.  Note  that  the  largest 
eastern  flank  of  v  is  uniciue. 
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flank 


B 


Figure  6.  Eastern  flank 

(2)  An  eastern  wing  of  v  is  a  subregion  R  Q  Rq  such  that  R  has  unique  ffW-  and  5W-comers 
and  has  its  AW-comer  lying  in  the  ray  projecting  vertically  downwards  from  the  NE- 
comer  of  v.  €i(v)  is  the  area  of  the  largest  eastern  wing  of  v.  Unlike  the  pyramid  aiid 
flank  above,  R  need  not  be  contained  in  M.  Note  that  since  an  eastern  flank  is  also  an 
eastern  wing,  we  have  €q(v)  s  ti(v). 
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Figure  7.  Eastern  wing 


(3)  Suppose  V  abuts  an  eastern  window  w.  A  hidden  eastern  wing  of  v  is  an  eastern  wing 
that  is  additionally  restricted  to  lie  within  the  pocket  outside  w.  Now  define  c^C^)  ^  ^^ 
area  of  the  largest  hidden  eastern  wing  of  v;  if  the  window  w  does  not  exist  then  define 
tjCv)  =  0.  Qearly  CjCv)  ^  ei(v). 


12 


hidden 


B 


Figure  8.  Hidden  eastern  wing 


(4)  An  eastern  chamber  of  v  is  a  subregion  o{  R  Q  Rq  such  that  R  has  a  unique  51V-comer 
and  its  westernmost  edge  contains  the  eastern  edge  of  v.  (.^(v)  is  the  area  of  the  largest 
eastern  wing  of  v.  Note  that  the  chamber  is  necessarily  contained  in  a  pocket  (and  thus 
completely  hidden);  again  C3(v)  =  0  if  v  does  not  abut  an  eastern  window. 


chamber 


B 


Figure  9.  Eastern  chamber 


(5)  ci),(v)  (j  =  0,  .  .    ,  3)  is  the  western  analogue  of  €,(v). 

(6)  3(v)  is  the  area  of  the  'rectangle  below  v',  ie.,  the  unique  largest  rectangle  in  M  whose 
N-edge  coincides  with  the  ^-edge  of  v. 
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rectangle  below  v 


B 


Figxire  10.  Rectangle  below  v 

(7)  For  lack  of  a  better  word,  a  subregion  R  is  called  v-centered  if  it  contains  the  rectangle 
/?'  below  V  and  R-R'  has  two  components  that  are  either  both  wings  or  a  wing  and  a 
chamber.  Note  that  the  components  be  empty  and  it  is  necessarily  the  case  that  one 
component  is  eastern  and  the  other  western.  Fmally,  a(v)  is  the  area  of  the  largest  v- 
centered  subregion. 

A  stage  of  the  algorithm  is  defined  as  the  period  during  which  a  particular  slab  is  being 
processed.   At  each  stage  of  the  algorithm,  we  have  a  set  S  of  slabs  of  Rq  such  that 

(a)  every  slab  processed  up  to  this  point  is  above  some  slab  in  B 

(b)  for  every  unprocessed  slab  B  that  is  ready  to  be  processed  next,  if  fl'  is  immediately 
above  B  then  B'  €  B. 

(Note:  the  slabs  in  B  need  not  be  pairwise  incomparable  with  respect  to  the  'above/below* 
relation.  To  see  this,  suppose  fl  in  S  has  two  slabs  B'  and  B"  immediately  below  it.  After 
processing  B' ,  we  now  have  B'  in  B  but  we  still  cannot  remove  B  from  B  until  B"  is 
processed.)  We  call  slabs  in  fl  the  base  slabs.  For  each  base  slab  B,  we  store  the  induced  slab 
tree  Tg.   At  each  node  v  in  T^  we  have  the  values  €(,  o),,  etc.,  described  above.   We  claim: 


Lemma  6.  By  the  time  all  the  slabs  of  Rq  are  processed,  the  area  of  a  maximum 
subregion  of  Rq  is  equal  to  the  largest  a(v)  eiKOuntered  over  the  course  of  the  algorithm. 

Proof.  Suppose  /?♦  is  a  maximum  subregion.  Consider  those  points  of  R*  that  are 
illuminated  b-  the  southernmost  edge  o{  R*:  among  these,  the  points  with  the  highest  altitude 
form  a  Jr-iizontal  line  segment  a  on  the  boundary  of  R*.  Define  ^  to  be  the  largest  rectangle 
in  R*  whose  A/-edge  coincides  with  ct.  Let  fl  be  the  slab  of  Rq  whose  5-edge  contains  the  5- 
edge  of  R.  Consider  the  induced  slab  tree  Tg  constructed  during  the  algorithm:  if  v  is  the 
node  of  Tg  whose  A^-edge  coincides  with  the  A^-edge  of  R  (note  that  v  is  well-defined)  then  we 
see  that  R*  is  indeed  a  v-centered  region.  So  by  definition,  a(v)  is  the  area  of  /?♦ .  Q.E.D 
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It  remains  to  show  how  to  maintain  the  values  described.  The  set  of  values 
{co(v),  cjo(v),  p(v)  :  V  6  Tg}  are  straightforward  to  compute  in  linear  time  from  Tg,  so  we  will 
not  explicitly  describe  how  to  compute  them.  Note  that  t,(v)  and  a),(v)  for  j  =  2,  3  are 
values  that  depend  only  on  the  pockets  outside  the  windows  of  the  skyline:  for  this  reason  we 
call  these  the  hidden  values.  Assuming  these  are  (recursively)  available,  we  can  compute  the 
rest  of  the  values  as  shown  next: 

L«nmu  7.  Let  Tg  be  the  induced  slab  tree  rooted  at  5.  If  the  values  e^Cv),  €3(v),  a>2(v) 
and  u)3(v)  for  each  node  v  in  T^  are  available  then  all  the  values  for  tj,  co,  and  a  can  be 
computed  in  0(n)  time. 

Proof.  The  value  €i(v)  (and  (Di(v)  similarly)  can  be  computed  by  the  following  formula: 

£i(v)  =  max{£o(v),  £2(v),  €i(v')  +  8(v)} 

where  v'  is  the  node  immediately  below  v,  8(v)  is  the  area  of  the  (possibly  degenerate) 
rectangle  whose  AW-comer  (resp.  A^£ -comer)  coincides  with  the  Sf -comer  (resp.  ^£ -comer) 
of  V  (resp.  v')  and  whose  5-edge  lies  in  the  5-edge  of  5.  (If  v  =  5  then  v'  is  undefined  but 
we  set  t,(v')  =  0  in  the  formula.)  To  justify  the  formula,  note  that  the  maximum  eastem 
wing  is  either  fully  illuminated  (cj  =  tg)  or  is  contained  in  the  eastem  pocket  abutting  v 
(cj  =  Ct)  or  is  entirely  south  and  east  of  the  5£ -comer  of  v.  This  last  case  gives  the  recursive 
expression  in  terms  of  ei(v')  and  the  area  S(v)  described  above. 

It  remains  to  indicate  a  formula  for  a(v): 

a(v)  =  p(v)  +  max{a)i(v)-(-€2(v),  a)2(v)-l-tj(v),  a)i(v)-l-ti(v)}. 

This  formula  is  almost  straight  from  the  definition  of  a.  Again  we  have  three  cases  for  the 
maximum  v-centered  subregion  R* :  the  first  case  is  where  the  northernmost  edge  of  R*  is  in 
an  eastem  chamber  of  v,  the  second  is  where  the  northernmost  edge  is  in  a  westem  chamber, 
and  the  last  case  is  where  the  northernmost  edge  is  (partly)  illuminated.  Q.E.D. 

Our  next  task  is  to  show  how  to  update  the  hidden  values  e,,  to,  (i  =  2,  3)  assumed  in 
the  above  lemma.  Updating  is  necessary  whenever  the  windows  of  the  Manhattan  skyline 
change,  more  precisely,  whenever  a  new  window  is  formed.  Suppose  B'  is  the  new  base  being 
processed  in  the  current  stage.  Fu^t  let  us  assume  that  B'  is  immediately  below  a  single  base 
B  i  B.  Then  a  new  eastem  window  is  formed  iff  the  eastem  edge  of  fl  is  east  of  the  eastem 
edge  oiB' .  For  simplicity,  let  us  first  assume  that  no  new  western  window  is  formed: 
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Figure  11.  Formation  of  a  new  eastern  window 

Let  w  be  the  new  eastern  window  (w  is  aligned  with  the  eastern  edge  of  B').  Let 
Vj,  .  .  .  ,  v^  be  the  slabs  of  Tg  that  intersect  w,  where  Vj  is  immediately  above  v^+i  and 
v^  =  5.  Let  w  split  Vj  into  a  western  slab  Uj  and  an  eastern  u\.  The  tree  Tg  is  then  split  into 
a  western  T  and  a  eastern  T .  Note  that  the  new  slab  tree  Tg-  is  just  T  appended  with  a  new 
root  B' ;  also  T  will  be  useful  for  the  computation  to  be  described  next  but  it  can  be  discarded 
after  that.  It  is  not  hard  to  see  that  using  any  reasonable  representation  (such  as  using 
pointers  for  edges)  for  Tg,  we  can  construct  T  and  T  from  Tg  in  linear  time.  Qearly  to 
compute  the  hidden  values  of  T  it  is  sufficient  to  compute  t^Cu/)  and  (.jiui)  for  i  ~  1 k. 

Consider  the  Manhattan  skyline  corresponding  to  T  (this  is  just  the  part  illuminated  by 
u\).  For  each  v  in  F,  the  values  €,(v)  (j  =  2,  3)  are  intact  from  Tg.  If  we  'seal'  off  the 
window  w  (ie.  treat  h-  as  if  it  were  part  of  the  boundary  of  /?q),  then  the  values  of  oj,(v) 
(i  =  2,  3)  are  also  available  for  all  v  €  T'.  By  the  previous  lemma,  the  other  values 
(a(v),  €o(v),  ti(v),  etc.)  can  be  computed  from  e^  and  (d,  («  =  2,  3)  in  linear  time.  It  is  easy 
to  see  that 

e2(«/)  =  3(«'/)  +  ei(u',). 


To  determine  f-iiuj),  we  proceed  as  follows.    For  each  v  €  T*,  it  is  easy  to  see  that  the 
largest  western  flank  R  ot  v  abuts  the  window  w.    Furthermore,  the  northern  enc^  point  of 

Jjnw  is  the  AAV-comer  of  some  u'j  (j  =  1 k):  we  say  v  belongs  to  u'j  in  th-.  case.    If  v 

has  an  empty  western  flank  then  v  is  one  of  the  u'y  and  we  say  v  belongs  to  iiself.   We  claim 
that  f.->,{ui)  is  the  maximum  value  of  the  expression 


€c(v)  +  p(v)  +  max{€i(v),€3(v)} 


(•) 
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where  v  ranges  over  the  nodes  belonging  to  u'j  for  some  j  &  /.  To  see  this,  let  ^*  be  a 
maximum  eastern  chamber  of  «(.  The  southernmost  edge  of  R*  is  contained  in  the  southern 
edge  of  u\.  As  in  the  proof  of  lemma  6,  consider  the  points  illuminated  by  the  southernmost 
edge  oi  R*:  the  subset  a  of  these  points  with  the  highest  altitude  forms  a  horizontal  segment. 
Let  V  be  the  node  of  T  whose  northern  edge  is  a  (this  is  well-defined).  Qearly  v  belongs  to 
some  u'j  with  j  ^  i-  The  area  of  ^*  is  given  by  the  expression  (*)  for  this  choice  of  v, 
proving  the  claim.  Computationally,  it  is  easy  to  partition  the  nodes  of  T  according  to  the  u', 
that  they  belong  to.  Then  for  each  j,  determine  the  largest  value  of  the  expression  (')  above 
among  those  v  belonging  to  some  j  s  i.  All  this  can  be  done  in  linear  time. 

This  completes  the  discussion  of  updating  the  hidden  values  of  each  slab  v  in  the  case 
where  one  new  window  is  formed  when  we  process  B' .  The  other  possibilities  are: 

(a)  The  current  slab  B'  is  not  below  any  other  slabs.  This  case  is  trivial. 

(b)  The  current  slab  B'  is  immediately  below  a  single  base  slab  fl  in  S  but  now  the  £-edge 
of  B'  is  west  of  the  £-edge  of  B,  and  the  W-edge  of  fl'  is  east  of  the  W-edge  of  B  (so  a 
new  eastern  and  new  western  window  are  formed).  Now  we  have  to  split  the  slab  tree 
Tg  into  three  trees  T"  ,T,  T  corresponding  to  the  eastern,  central  and  western  portion 
of  the  split,  respectively.  We  can  apply  the  previous  method  using  T  to  compute  the  e, 
values  for  the  new  eastern  window,  and  T'  for  the  western  window. 

(c)  The  current  slab  B'  is  immediately  below  two  or  more  base  slabs  in  B.  At  most  two 
windows  are  formed  and  this  can  be  treated  as  in  (a). 

We  have  now  completely  described  the  algorithm  and  conclude  with  our  main  result. 

Theorem  8.  The  maximum  1-concave  subregion  of  a  simple  rectilinear  region  can  be 
found  in  0{n^)  time. 

Proof.  Using  the  above  algorithm,  there  are  0{n)  slabs  to  process  and  each  slab  can  be 
processed  in  linear  time.  The  algorithm  as  described  only  keeps  track  of  the  area  but  it  is  not 
hard  to  modify  them  to  remember  the  actual  subregion  with  the  remembered  area. 
Alternatively,  we  can  proceed  in  two  stages  as  follows:  In  the  first  stage,  proceed  as  described 
except  that  we  also  keep  track  of  the  extremal  N-,  E-  and  W-edge  of  the  various  maximum  1- 
concave  regions  (flanks,  wings,  etc.)  connected  with  each  of  the  numbers  t,(v),  etc.  At  the 
end  of  stage  1,  we  have  the  extremal  edges  of  the  maximum  subregion.  It  is  now  quite  easy 
to  find  the  connecting  staircases  between  adjacent  extremal  edges  in  0(n  log  n)  time,  a 

We  remark  that  the  algorithm  still  works  correctly  and  the  time-bound  remains  valid 
even  when  the  given  rectilinear  region  has  holes. 
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